🕷️️ Job Radar • SCRAPING

freelancer.com 🟢 2026-05-03

🔹 Web-Scraping Setup for Product Details from Australian E-commerce Sites
👤 Client: 🇦🇺 Sydney, Australia Member since 2024-11-25
💰 Price: $17 Average bid
🚩 Problem: Need a reliable web-scraping solution to gather product details, recognize duplicates, and consolidate into a Shopify-ready CSV.
📦 Existing: Not specified

Specifications:

[Target] Specific e-commerce, retailer, and manufacturer websites in Australia
[Method] Python with Scrapy for scraping, BeautifulSoup for parsing HTML, Selenium for dynamic content handling
[UI/UX] Not applicable (backend task)
[Stack] Python, Scrapy, BeautifulSoup, Selenium, Pandas for data processing
[Security] Secure login handling and data encryption during transmission
[Format] CSV formatted to Shopify’s column requirements

Workflow:

1. Define the list of target websites and their URLs.
2. Develop a Scrapy spider with custom middleware for handling dynamic content using Selenium.
3. Parse HTML content using BeautifulSoup to extract product details (name, price, kit configuration, shipping cost, subscription pricing).
4. Implement deduplication logic based on product name and other unique identifiers.
5. Store scraped data in a structured format with source site information for each record.
6. Use Pandas to clean and transform the data into a CSV file formatted according to Shopify’s requirements.
7. Test the scraper by running an initial full pull of sample products from different sites.
8. Document the codebase, including setup instructions and regional site list management.